This script takes the raw outputs from the water quality sensors at Falling Creek and combines them into one file, identifies gaps in the data, runs the data through the QAQC function (qaqc_fcr), and produces QAQC plots for visual inspection.

  1. Open the script and check you have the correct function in line 19
  2. Change the current start and end times to match the year you are publishing for
  3. If there are more than one csv make sure they are all in the Download data chunk. Currently there is only one csv with all of the observations.
  4. Once you have made sure everything is up to date press the knit button and it will produce an html file with all of the outputs and plots.

Use this file path in Carey Lab Reservoir GitHub Repo: “./Data/DataAlreadyUploadedToEDI/EDIProductionFiles/MakeEML_FCRcatwalk/2023/”

Check for gaps in the data frame

This identifies if there are any daily data gaps in the long-term record

This identifies if there are any sub-daily gaps in the long-term record. Most of these gaps happen when we change the program on the data logger.

## [1] "2018-07-04 13:50:00 UTC" "2018-07-05 13:50:00 UTC"
## [1] "2018-08-06 09:00:00 UTC" "2018-08-06 10:00:00 UTC"
## [1] "2018-08-06 10:00:00 UTC" "2018-08-06 10:10:00 UTC"
## [1] "2018-08-06 10:10:00 UTC" "2018-08-06 10:30:00 UTC"
## [1] "2018-08-06 10:30:00 UTC" "2018-08-06 11:10:00 UTC"
## [1] "2018-08-06 11:10:00 UTC" "2018-08-06 11:40:00 UTC"
## [1] "2018-08-06 11:40:00 UTC" "2018-08-06 12:00:00 UTC"
## [1] "2018-08-27 12:10:00 UTC" "2018-08-27 16:50:00 UTC"
## [1] "2018-08-28 11:00:00 UTC" "2018-08-28 14:50:00 UTC"
## [1] "2019-04-15 09:40:00 UTC" "2019-04-15 10:10:00 UTC"
## [1] "2019-04-15 10:10:00 UTC" "2019-04-15 10:30:00 UTC"
## [1] "2019-04-15 10:30:00 UTC" "2019-04-15 10:50:00 UTC"
## [1] "2019-04-15 10:50:00 UTC" "2019-04-15 11:10:00 UTC"
## [1] "2019-04-15 11:10:00 UTC" "2019-04-15 11:30:00 UTC"
## [1] "2019-05-20 12:20:00 UTC" "2019-05-20 12:30:00 UTC"
## [1] "2020-05-15 15:00:00 UTC" "2020-05-15 15:10:00 UTC"
## [1] "2020-10-09 08:00:00 UTC" "2020-10-09 09:20:00 UTC"
## [1] "2020-10-19 14:00:00 UTC" "2020-10-19 17:00:00 UTC"
## [1] "2020-10-20 10:00:00 UTC" "2020-10-20 10:40:00 UTC"
## [1] "2020-10-20 11:00:00 UTC" "2020-10-20 11:10:00 UTC"
## [1] "2020-10-20 12:00:00 UTC" "2020-10-20 12:10:00 UTC"
## [1] "2020-10-20 13:00:00 UTC" "2020-10-20 13:40:00 UTC"
## [1] "2020-10-22 09:00:00 UTC" "2020-10-22 09:30:00 UTC"
## [1] "2021-01-07 10:20:00 UTC" "2021-01-07 10:30:00 UTC"
## [1] "2021-01-07 11:00:00 UTC" "2021-01-07 11:30:00 UTC"
## [1] "2021-02-26 11:00:00 UTC" "2021-02-26 11:02:00 UTC"
## [1] "2021-02-26 12:00:00 UTC" "2021-02-26 12:40:00 UTC"
## [1] "2021-06-28 13:00:00 UTC" "2021-06-28 14:00:00 UTC"
## [1] "2021-08-17 06:00:00 UTC" "2021-08-17 06:10:00 UTC"
## [1] "2022-05-02 13:20:00 UTC" "2022-05-02 13:30:00 UTC"
## [1] "2022-05-05 08:30:00 UTC" "2022-05-05 08:50:00 UTC"
## [1] "2023-01-24 14:00:00 UTC" "2023-01-24 14:10:00 UTC"
## [1] "2023-02-03 13:50:00 UTC" "2023-02-03 14:10:00 UTC"
## [1] "2023-02-07 11:50:00 UTC" "2023-02-07 12:00:00 UTC"
## [1] "2023-03-07 12:00:00 UTC" "2023-03-07 12:20:00 UTC"
## [1] "2023-04-18 11:40:00 UTC" "2023-04-18 13:30:00 UTC"
## [1] "2023-05-29 07:20:00 UTC" "2023-05-29 11:00:00 UTC"
## [1] "2023-06-26 11:10:00 UTC" "2023-06-26 12:10:00 UTC"
## [1] "2023-07-19 09:20:00 UTC" "2023-07-19 10:20:00 UTC"
## [1] "2023-07-31 12:50:00 UTC" "2023-07-31 14:40:00 UTC"
## [1] "2023-08-07 12:20:00 UTC" "2023-08-07 12:30:00 UTC"

Read in the QAQC File and Check it out

This section reads in the QAQC file and then you can look at the head, tail and structure. Make sure the last row is Dec. 31 23:50 of the publishing year. There should be 83 columns unless a new one has been added.

## Rows: 272972 Columns: 83
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr   (1): Reservoir
## dbl  (81): Site, ThermistorTemp_C_surface, ThermistorTemp_C_1, ThermistorTem...
## dttm  (1): DateTime
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
## # A tibble: 6 × 83
##   Reservoir  Site DateTime            ThermistorTemp_C_surf…¹ ThermistorTemp_C_1
##   <chr>     <dbl> <dttm>                                <dbl>              <dbl>
## 1 FCR          50 2018-07-05 13:50:00                    29.0               27.9
## 2 FCR          50 2018-07-05 14:00:00                    29.2               28.0
## 3 FCR          50 2018-07-05 14:10:00                    29.3               28.1
## 4 FCR          50 2018-07-05 14:20:00                    30.1               28.1
## 5 FCR          50 2018-07-05 14:30:00                    29.8               28.2
## 6 FCR          50 2018-07-05 14:40:00                    29.6               28.3
## # ℹ abbreviated name: ¹​ThermistorTemp_C_surface
## # ℹ 78 more variables: ThermistorTemp_C_2 <dbl>, ThermistorTemp_C_3 <dbl>,
## #   ThermistorTemp_C_4 <dbl>, ThermistorTemp_C_5 <dbl>,
## #   ThermistorTemp_C_6 <dbl>, ThermistorTemp_C_7 <dbl>,
## #   ThermistorTemp_C_8 <dbl>, ThermistorTemp_C_9 <dbl>, RDO_mgL_5 <dbl>,
## #   RDOsat_percent_5 <dbl>, RDO_mgL_5_adjusted <dbl>,
## #   RDOsat_percent_5_adjusted <dbl>, RDOTemp_C_5 <dbl>, RDO_mgL_9 <dbl>, …
## # A tibble: 6 × 83
##   Reservoir  Site DateTime            ThermistorTemp_C_surf…¹ ThermistorTemp_C_1
##   <chr>     <dbl> <dttm>                                <dbl>              <dbl>
## 1 FCR          50 2023-09-18 23:50:00                    21.6               21.6
## 2 FCR          50 2023-09-19 00:00:00                    21.6               21.6
## 3 FCR          50 2023-09-19 00:10:00                    21.6               21.5
## 4 FCR          50 2023-09-19 00:20:00                    21.5               21.5
## 5 FCR          50 2023-09-19 00:30:00                    21.5               21.5
## 6 FCR          50 2023-09-19 00:40:00                    21.5               21.5
## # ℹ abbreviated name: ¹​ThermistorTemp_C_surface
## # ℹ 78 more variables: ThermistorTemp_C_2 <dbl>, ThermistorTemp_C_3 <dbl>,
## #   ThermistorTemp_C_4 <dbl>, ThermistorTemp_C_5 <dbl>,
## #   ThermistorTemp_C_6 <dbl>, ThermistorTemp_C_7 <dbl>,
## #   ThermistorTemp_C_8 <dbl>, ThermistorTemp_C_9 <dbl>, RDO_mgL_5 <dbl>,
## #   RDOsat_percent_5 <dbl>, RDO_mgL_5_adjusted <dbl>,
## #   RDOsat_percent_5_adjusted <dbl>, RDOTemp_C_5 <dbl>, RDO_mgL_9 <dbl>, …
## tibble [272,972 × 83] (S3: tbl_df/tbl/data.frame)
##  $ Reservoir                    : chr [1:272972] "FCR" "FCR" "FCR" "FCR" ...
##  $ Site                         : num [1:272972] 50 50 50 50 50 50 50 50 50 50 ...
##  $ DateTime                     : POSIXct[1:272972], format: "2018-07-05 13:50:00" "2018-07-05 14:00:00" ...
##  $ ThermistorTemp_C_surface     : num [1:272972] 29 29.2 29.3 30.1 29.8 ...
##  $ ThermistorTemp_C_1           : num [1:272972] 27.9 28 28.1 28.1 28.2 ...
##  $ ThermistorTemp_C_2           : num [1:272972] 24.6 24.5 24.3 24.4 24.5 ...
##  $ ThermistorTemp_C_3           : num [1:272972] 17.6 17.7 17.8 17.9 17.8 ...
##  $ ThermistorTemp_C_4           : num [1:272972] 13.6 13.6 13.8 13.9 14 ...
##  $ ThermistorTemp_C_5           : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
##  $ ThermistorTemp_C_6           : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
##  $ ThermistorTemp_C_7           : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
##  $ ThermistorTemp_C_8           : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
##  $ ThermistorTemp_C_9           : num [1:272972] 11.1 11.1 11.1 11.1 11.1 ...
##  $ RDO_mgL_5                    : num [1:272972] 7.61 7.58 7.91 7.87 7.71 ...
##  $ RDOsat_percent_5             : num [1:272972] 69.4 69.1 72.1 71.8 70.3 ...
##  $ RDO_mgL_5_adjusted           : num [1:272972] 7.61 7.58 7.91 7.87 7.71 ...
##  $ RDOsat_percent_5_adjusted    : num [1:272972] 69.4 69.1 72.1 71.8 70.3 ...
##  $ RDOTemp_C_5                  : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
##  $ RDO_mgL_9                    : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ RDOsat_percent_9             : num [1:272972] 70.7 70.6 70.7 70.7 70.7 ...
##  $ RDO_mgL_9_adjusted           : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ RDOsat_percent_9_adjusted    : num [1:272972] 70.7 70.6 70.7 70.7 70.7 ...
##  $ RDOTemp_C_9                  : num [1:272972] 11.2 11.2 11.2 11.2 11.2 ...
##  $ EXOTemp_C_1                  : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOCond_uScm_1               : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOSpCond_uScm_1             : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOTDS_mgL_1                 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXODOsat_percent_1           : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXODO_mgL_1                  : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOChla_RFU_1                : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOChla_ugL_1                : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOBGAPC_RFU_1               : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOBGAPC_ugL_1               : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOfDOM_RFU_1                : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOfDOM_QSU_1                : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOTurbidity_FNU_1           : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOPressure_psi              : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXODepth_m                   : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOBattery_V                 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOCablepower_V              : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ EXOWiper_V                   : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ LvlPressure_psi_9            : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ LvlTemp_C_9                  : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ LvlDepth_m_9                 : num [1:272972] NA NA NA NA NA NA NA NA NA NA ...
##  $ RECORD                       : num [1:272972] 0 1 2 3 4 5 6 7 8 9 ...
##  $ CR6Battery_V                 : num [1:272972] 13.3 13.2 13.2 13.2 13.3 ...
##  $ CR6Panel_Temp_C              : num [1:272972] 33.4 34 34.6 35.2 35.6 ...
##  $ Flag_ThermistorTemp_C_surface: num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_1      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_2      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_3      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_4      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_5      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_6      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_7      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_8      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_ThermistorTemp_C_9      : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_RDO_mgL_5               : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_RDOsat_percent_5        : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_RDOTemp_C_5             : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_RDO_mgL_9               : num [1:272972] 2 2 2 2 2 2 2 2 2 2 ...
##  $ Flag_RDOsat_percent_9        : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_RDOTemp_C_9             : num [1:272972] 0 0 0 0 0 0 0 0 0 0 ...
##  $ Flag_EXOTemp_C_1             : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOCond_uScm_1          : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOSpCond_uScm_1        : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOTDS_mgL_1            : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXODOsat_percent_1      : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXODO_mgL_1             : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOChla_RFU_1           : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOChla_ugL_1           : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOBGAPC_RFU_1          : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOBGAPC_ugL_1          : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOfDOM_RFU_1           : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOfDOM_QSU_1           : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOTurbidity_FNU_1      : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOPressure_psi         : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXODepth_m              : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOBattery_V            : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOCablepower_V         : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_EXOWiper_V              : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_LvlPressure_psi_9       : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...
##  $ Flag_LvlTemp_C_9             : num [1:272972] 7 7 7 7 7 7 7 7 7 7 ...

Let’s look at the flag Frequency for each variable. As a reminder here are the flag codes Flag values

0: no flag

1: value removed due to maintenance and set to NA

2: negative or outlier value removed and set to NA, see Methods section for more detail on QAQC process

3: negative values set to 0

4: value removed due to fouling and set to NA

5: questionable value due to potential fouling

6: very questionable value due to potential fouling. Values adjusted using a linear or square root function to match high-resolution CTD profiles are given in RDO_mgL_5 and RDO_sat_percent_5

7: missing data

8: Value corrected using a constant offset due to two thermistor malfunctions in Fall 2020

## Flag_ThermistorTemp_C_surface
##      0      1      5      7   <NA> 
## 269512   2013      1   1446      0 
## Flag_ThermistorTemp_C_1
##      0      1      5      7      8   <NA> 
## 252526   4472      1   1446  14527      0 
## Flag_ThermistorTemp_C_2
##      0      1      5      7   <NA> 
## 269361   2164      1   1446      0 
## Flag_ThermistorTemp_C_3
##      0      1      5      7   <NA> 
## 269620   1905      1   1446      0 
## Flag_ThermistorTemp_C_4
##      0      1      5      7      8   <NA> 
## 252622   4472      1   1446  14431      0 
## Flag_ThermistorTemp_C_5
##      0      1      5      7   <NA> 
## 268908   2617      1   1446      0 
## Flag_ThermistorTemp_C_6
##      0      1      5      7   <NA> 
## 268908   2617      1   1446      0 
## Flag_ThermistorTemp_C_7
##      0      1      5      7   <NA> 
## 268908   2617      1   1446      0 
## Flag_ThermistorTemp_C_8
##      0      1      5      7   <NA> 
## 268908   2617      1   1446      0 
## Flag_ThermistorTemp_C_9
##      0      1      5      7   <NA> 
## 268908   2617      1   1446      0 
## Flag_RDO_mgL_5
##      0      1      2      3      5      6      7   <NA> 
## 216159   3624     14  37437  12950    220   2568      0 
## Flag_RDOsat_percent_5
##      0      1      2      3      5      6      7   <NA> 
## 216147   3624     26  37437  12950    220   2568      0 
## Flag_RDOTemp_C_5
##      0      1      2      5      7   <NA> 
## 268038   2335     12      1   2586      0 
## Flag_RDO_mgL_9
##      0      1      2      3      5      6      7   <NA> 
## 193906   4082   6911  51428  12303   3214   1128      0 
## Flag_RDOsat_percent_9
##      0      1      2      3      5      6      7   <NA> 
## 198241   4082    293  53710  12303   3214   1129      0 
## Flag_RDOTemp_C_9
##      0      1      5      7   <NA> 
## 269204   2617      1   1150      0 
## Flag_EXOTemp_C_1
##      0      1      2      5      7   <NA> 
## 258543   5441     13      1   8974      0 
## Flag_EXOCond_uScm_1
##      0      1      2      5      7   <NA> 
## 255166   5420   3388     44   8954      0 
## Flag_EXOSpCond_uScm_1
##      0      1      2      5      7   <NA> 
## 255165   5420   3389     44   8954      0 
## Flag_EXOTDS_mgL_1
##      0      1      2      5      7   <NA> 
## 255165   5420   3389     44   8954      0 
## Flag_EXODOsat_percent_1
##      0      1      2      3      5      7   <NA> 
## 256282   6718     19   1028      1   8924      0 
## Flag_EXODO_mgL_1
##      0      1      2      3      5      7   <NA> 
## 254157   6718     19   1021      1  11056      0 
## Flag_EXOChla_RFU_1
##      0      1      2      3      4      5      7   <NA> 
## 256229   5441    175      7      6   2140   8974      0 
## Flag_EXOChla_ugL_1
##      0      1      2      4      5      7   <NA> 
## 256322   5441    166      5   2064   8974      0 
## Flag_EXOBGAPC_RFU_1
##      0      1      2      3      5      7   <NA> 
## 255752   5441     17    672   2116   8974      0 
## Flag_EXOBGAPC_ugL_1
##      0      1      2      3      5      7   <NA> 
## 252992   5441     18   3581   1966   8974      0 
## Flag_EXOfDOM_RFU_1
##      0      1      2      3      5      7   <NA> 
## 256999   6882    117     19      1   8954      0 
## Flag_EXOfDOM_QSU_1
##      0      1      2      3      5      7   <NA> 
## 256998   6882    118     19      1   8954      0 
## Flag_EXOTurbidity_FNU_1
##      0      1      2      3      5      7   <NA> 
##  57611  17768   1127     24      1 196441      0 
## Flag_EXOPressure_psi
##      0      1      2      5      7   <NA> 
## 258548   5441      8      1   8974      0 
## Flag_EXODepth_m
##      0      1      2      5      7   <NA> 
## 258548   5441      8      1   8974      0 
## Flag_EXOBattery_V
##      0      1      2      5      7   <NA> 
## 258555   5441      1      1   8974      0 
## Flag_EXOCablepower_V
##      0      1      2      5      7   <NA> 
## 258539   5441     17      1   8974      0 
## Flag_EXOWiper_V
##      0      1      2      5      7   <NA> 
## 258124   5441    432      1   8974      0 
## Flag_LvlPressure_psi_9
##      0      1      5      7   <NA> 
## 165746   2617      1 104608      0 
## Flag_LvlTemp_C_9
##      0      1      5      7   <NA> 
## 165746   2617      1 104608      0

QAQC Plots

Temperature

Depth

Dissolved Oxygen

Chlorophyll

Phycocanin

fDOM

Conductivity, Specific Conductivity, TDS

Turbidity